Incorporating Medical Student Documentation Into the Billable Encounter: A Pragmatic Approach to Implementation of the 2018 Centers for Medicare & Medicaid Services Rule Revision.

In early 2018, the Centers for Medicare & Medicaid Services released the Medical Review of Evaluation and Management (E/M) Documentation, which allows supervising teaching physicians to rely on a medical student's documentation to support billing for E/M services. This change has potential to enhance education, clinical documentation quality, and the satisfaction of students, postgraduate trainees, and teaching physicians. However, its practical adoption presents many challenges that must be navigated successfully to realize these important goals in compliance with federal and local requirements, while avoiding unintended downstream problems. Implementation requires careful planning, policy creation, education, and monitoring, all with collaboration between institutional leaders, compliance and information technology professionals, educators, and learners. In this paper, we review the 2018 Centers for Medicare & Medicaid Services rule change, address common questions and potential impacts, outline practical workflows to meet the supervision requirement, and discuss steps for successful implementation.

Impact of peer assessment on the professional development of medical students: a qualitative study.

PURPOSE: Peer assessment can predict future academic performance and provide medical students with reliable feedback about professionalism. It is unclear whether peer assessment fosters personal growth or transformations in attitudes or behaviors. The authors investigated what types of peer feedback students remember and what reactions or transformations students experience as a result of peer assessment. METHOD: In May 2005, the authors invited medical students from the second-year (n = 101) and fourth-year (n = 83) classes to provide narratives about how peer assessment affected their personal and professional development. All students had participated in peer assessment during required, formative comprehensive assessments. The authors analyzed responses using mixed qualitative-quantitative methods. RESULTS: The 138 responses represented 82% and 69% of students from the fourth-year and second-year classes, respectively. Students recalled peer assessment content about both positive (e.g., teaching skills) and negative (e.g., overconfidence) qualities. Both positive (e.g., appreciation) and negative (e.g., anger) emotional reactions were reported. Many (67%) found peer assessment helpful, reassuring, or confirming of something they knew; 65% reported important transformations in awareness, attitudes, or behaviors because of peer assessment. Change was more likely when feedback was specific and described an area for improvement. Wholly negative responses to the peer assessment process were rare. CONCLUSIONS: Peer assessment can be a powerful tool to assess and encourage formation of professional behaviors, particularly the interpersonal dimensions. Participants should receive training to provide specific, constructive feedback. The institutional culture should emphasize safety around feedback, while committing to rewarding excellence and addressing concerning behaviors.

Further challenges in measuring communication skills: accounting for actor effects in standardised patient assessments.

CONTEXT: Subjective rating scales for communication skills may yield more personally meaningful responses than more standardised rating schemes. It is unclear, however, whether such evaluations may be overly biased by respondents' rating styles, which may lead to unreliable measurement of examinees' communication skills. METHODS: Our study involved 212 students from the classes of 2005 and 2006 at the University of Rochester School of Medicine and Dentistry. All students were rated by actors depicting standardised patients (SPs) on the same seven cases using the 19-item Rochester Communication Rating Scale (RCRS). Different students were assigned to different actors playing the same SP. We assessed the extent to which actors' personal rating styles influenced the scores they assigned to students. Main outcome measures were: between-actor variability in responses; the degree to which actors' response styles contribute to overall scores, and improvements in reliability achieved by standardising actors' ratings. RESULTS: There were statistically significant differences between actors in their mean assigned scores. Scores aggregated over 18 separate SP cases have an expected generalisability coefficient of 0.79. If raw RCRS scores are used, a total of 27 replications of the RCRS are required to achieve a Cronbach's alpha of 0.8; standardisation reduces this number to 18. CONCLUSIONS: Although actors are variable in their use of a standardised subjective scale of communication, such differences contribute to an acceptably small proportion of the total variance if scores are combined across a large number of cases. Reliability can be markedly improved by standardising scores across raters.

Changes in self-perceived abilities among male and female medical students after the first year of clinical training.

BACKGROUND: It is known that male and female medical students have different experiences in their clinical training. AIMS: To assess whether male and female medical students change in their self-rated work habits and interpersonal habits during the first year of clinical training. METHOD: Longitudinal study of self- and peer-assessment among 224 medical students in 3 consecutive classes at a private US medical school. Students rated themselves on global work habits (WH) and interpersonal attributes (IA). Students also rated and were rated by 6-12 peers on the same scale. RESULTS: In the second year of medical school, there were no differences between men and women in quartiles of self-assessed WH or IA. At the end of the third year, however, women were more likely to be in the lower quartiles of self-assessed WH (X(2) = 6.77; p = 0.03), as well as the highest quartiles of self-assessed IA (X(2) = 11.36; p = 0.003). In both years, women rated their own WH skills significantly lower than they rated their peers, while men rated themselves similarly to peers. There were no sex differences in self-assessed IA. CONCLUSIONS: Although second-year male and female medical students appear similar to one another in terms of self-assessed WH and IA, by the end of the third year women rate themselves relatively lower in WH, while men rate themselves relatively lower in IA.

Relationship between peer assessment during medical school, dean's letter rankings, and ratings by internship directors.

BACKGROUND: It is not known to what extent the dean's letter (medical student performance evaluation [MSPE]) reflects peer-assessed work habits (WH) skills and/or interpersonal attributes (IA) of students. OBJECTIVE: To compare peer ratings of WH and IA of second- and third-year medical students with later MSPE rankings and ratings by internship program directors. DESIGN AND PARTICIPANTS: Participants were 281 medical students from the classes of 2004, 2005, and 2006 at a private medical school in the northeastern United States, who had participated in peer assessment exercises in the second and third years of medical school. For students from the class of 2004, we also compared peer assessment data against later evaluations obtained from internship program directors. RESULTS: Peer-assessed WH were predictive of later MSPE groups in both the second (F = 44.90, P < .001) and third years (F = 29.54, P < .001) of medical school. Interpersonal attributes were not related to MSPE rankings in either year. MSPE rankings for a majority of students were predictable from peer-assessed WH scores. Internship directors' ratings were significantly related to second- and third-year peer-assessed WH scores (r = .32 [P = .15] and r = .43 [P = .004]), respectively, but not to peer-assessed IA. CONCLUSIONS: Peer assessment of WH, as early as the second year of medical school, can predict later MSPE rankings and internship performance. Although peer-assessed IA can be measured reliably, they are unrelated to either outcome.

Effects of rater selection on peer assessment among medical students.

CONTEXT: Although peer-assessment appears promising as a method to assess interpersonal skills among medical students, results may be biased by method of peer selection, particularly if different kinds of classmates are assigned systematically by different methods. It is also unclear whether students with lower interpersonal skills may be more negative towards their classmates than students with higher levels of interpersonal skills and, if so, how much bias this may introduce into the results of peer assessment. It is also unclear whether low-rated students are more likely to ask to rate one another. DESIGN, SETTING, AND PARTICIPANTS: Participants were 296 medical students from the classes of 2004, 2005 and 2006 at the University of Rochester School of Medicine and Dentistry. All participants were evaluated by between 6 and 12 classmates near the end of both their second and third years. In some classes peer-raters were assigned from class lists, while in other classes students were asked to recommend classmates to rate and be rated by. MAIN OUTCOME MEASURES: Mean numerical ratings of interpersonal-attributes scores received from peers, compared with mean scores simultaneously assigned to peers. We also examined students' selection of peer-raters, as well as scores received by their peer-raters. RESULTS: For all 3 classes in both second and third years, students in the lowest quartile of received scores were significantly more likely to simultaneously assign lower scores to their peers. Among third-year students, such low-scoring students were significantly more likely to ask to rate and be rated by other low-scoring students. Among third-year students, low-rated students were more likely to rate and be rated by other low-rated students, irrespective of how rater assignments are made. Most students who were originally in the lowest quartile of received scores remained so after removing the scores assigned by low-rated raters. Among pre-clinical second-year students, however, low-rated students were no more likely than high-rated students to be rated by other low-rated students. CONCLUSION: Students with low levels of peer-assessed interpersonal attributes are more negative in their judgements of classmates. Third-year medical students with low levels of interpersonal attributes are more likely to be rated by other students with low levels of interpersonal attributes, irrespective of method of rater assignment. This effect was not seen in the second year of medical school. Biases in selection of raters among third-year students, however, do not appear to affect the results of peer assessment. Thus, medical schools that choose to institute peer assessment can select the method of peer-rater assignment based on practicality and logistical considerations. Students' affiliations with one another in the third year appear to be related to peer-assessed interpersonal attributes.

Temporal and group-related trends in peer assessment amongst medical students.

Peer assessment has been increasingly recommended as a way to evaluate the professional competencies of medical trainees. Prior studies have only assessed single groups measured at a single timepoint. Thus, neither the longitudinal stability of such ratings nor differences between groups using the same peer-assessment instrument have been reported previously. Participants were all members of 2 consecutive classes of medical students (n = 77 and n = 85) at the University of Rochester School of Medicine and Dentistry who completed Years 2 and 3 of medical school consecutively. All participants were evaluated by 6-12 classmates near the end of both Years 2 and 3. Main outcome measures were mean numerical ratings on peer-assessed scales of professional work habits (WH) and interpersonal attributes (IA). Both scales had high internal consistencies in both years (Cronbach's alpha 0.84-0.94). The IA and WH scales were moderately correlated with one another (r = 0.36 in Year 2, r = 0.28 in Year 3). Year 2 scores were predictive of Year 3 scores for both scales (WH: r = 0.64; IA; r = 0.62). Generalisability and decision analyses revealed that 1 class was consistently more discriminating with the WH scale, while the other was more discriminating with of the IA scale. Depending on the class, year and scale, the number of raters needed to achieve a reasonable reliability ranged between 7 and 28. Although Year 3 peer ratings were consistently higher than Year 2 peer ratings for both WH and IA, individual scores were highly correlated across the 2 years, despite the fact that different individuals were chosen as peer raters. Abilities appear to be stable between Years 2 and 3 of medical school. Groups may differ in their ability to discriminate different kinds of skills. Generalisability analysis can be used to discover these patterns within groups.

Peer assessment of professional competence.

BACKGROUND: Current assessment formats for medical students reliably test core knowledge and basic skills. Methods for assessing other important domains of competence, such as interpersonal skills, humanism and teamwork skills, are less well developed. This study describes the development, implementation and results of peer assessment as a measure of professional competence of medical students to be used for formative purposes. METHODS: Year 2 medical students assessed the professional competence of their peers using an online assessment instrument. Fifteen randomly selected classmates were assigned to assess each student. The responses were analysed to determine the reliability and validity of the scores and to explore relationships between peer assessments and other assessment measures. RESULTS: Factor analyses suggest a 2-dimensional conceptualisation of professional competence: 1 factor represents Work Habits, such as preparedness and initiative, and the other factor represents Interpersonal Habits, including respect and trustworthiness. The Work Habits factor had moderate, yet statistically significant correlations ranging from 0.21 to 0.53 with all other performance measures that were part of a comprehensive assessment of professional competence. Approximately 6 peer raters were needed to achieve a generalisability coefficient of 0.70. CONCLUSIONS: Our findings suggest that it is possible to introduce peer assessment for formative purposes in an undergraduate medical school programme that provides multiple opportunities to interact with and observe peers.

Comprehensive assessment of professional competence: the Rochester experiment.

BACKGROUND: A required 2-week comprehensive assessment (CA) for 2nd-year medical students that integrates basic science, clinical skills, information management, and professionalism was implemented. DESCRIPTION: The CA links standardized patients (SPs) with computer-based exercises, a teamwork exercise, and peer assessments; and culminates in student-generated learning plans. EVALUATION: Scores assigned by SPs showed acceptable interrater reliability. Factor analyses defined meaningful subscales of the peer assessment and communication rating scales. Ratings of communication skills were correlated with information gathering, patient counseling, and peer assessments; these, in turn, were strongly correlated with the written exercises. Students found the CA fair, with some variability in opinion of the peer and written exercises. Useful learning plans and positive curricular changes were undertaken in response to the CA results. CONCLUSION: A CA that integrates multiple domains of professional competence is feasible, useful to students, and fosters reflection and change. Preliminary data suggest that this format is reliable and valid.